1. Field of the Invention
The present invention relates to a morphing image processing system and a morphing processing method for performing morphing processing where intermediate images interpolated between a first image and a second image are formed so that the first image is deformed smoothly into the second image. In particular, the present invention relates to a morphing processing image processing system and method that represents an image with a polygon mesh and uses processing for reducing the number of polygons (polygon reduction).
2. Description of the Related Art
With high performance of the recent computer systems and development of multimedia processing techniques, an environment is emerging where a high level of three dimensional (hereinafter, abbreviated as 3D) computer graphics (hereinafter, abbreviated as CG) can be processed on personal computers or the like. Moreover, accelerator boards dedicated to processing of three dimensional CG is being provided. The image processing technique has been developed so that various special effects can be provided or synthesis processing can be performed to an image to edit the image.
In the current image processing, one of the noted techniques is morphing processing. Morphing processing is a technique of slowly and smoothly deforming a first image, which is the original image, into a second image, which is the target image. For example, this technique is used for video image special effects in movies, such as an effect used in a scene where a human is deformed gradually into a wolf.
In this morphing processing technique, multiple intermediate images interpolated between the first image and the second image are formed based on these images, and the intermediate images are displayed in the order of progression of the change so that the image can be deformed smoothly into the second image. The morphing processing is generally classified into two-dimensional morphing (hereinafter, referred to as 2D morphing), and three-dimensional morphing (hereinafter, referred to as 3D morphing). FIGS. 20A to 20C show the concept of 2D morphing processing.
First, data of a 2D image before deformation, which is the first image, and data of a 2D image after deformation, which is the second image are prepared. Next, points characterizing these images (hereinafter, referred to as control points) are selected from these 2D image data. In FIGS. 20A to 20C, an example of the control point of the first image is represented by AX, and an example of the control point of the second image is represented by ZX. In this example, the points having the same argument X correspond to each other. Next, the difference between the control point of the first image and the corresponding control point of the second image is calculated to obtain an intermediate value based on the difference, and further intermediate values are obtained to make the difference progressively smaller and smaller by a predetermined number of steps. Herein, obtaining the intermediate values in this manner is referred to as interpolation. Next, the intermediate values of the control points are collected at each step, and an image defined by these control points is restored. Then, an intermediate image represented by these control points can be obtained at each step. Then, the obtained intermediate images are displayed in the order of deformation in time series, so that a special effect of the first image being deformed gradually into the second image can be obtained.
In the above-described morphing processing, in order to achieve more smooth deformation of the image, a large number of control points are selected densely over the entire image, or the number of deformation steps is made large.
The 3D morphing is performed substantially in the same manner as in the 2D morphing. However, the 3D morphing is more complicated than the 2D morphing, because the control points are three-dimensionally selected.
The above-described morphing processing in the related art has the following problems.
A first problem is as follows. Since it is necessary to select a large number of control points densely to obtain a smoothly deforming image, and to perform processing at a large number of deformation steps, interpolation calculation of the control points is performed in a great amount, resulting in complicated processing and increased time-cost. The interpolation calculation amount can be reduced simply by reducing the number of the control points to be selected. In this case, however, the image is distorted during deformation, resulting in poor image quality. Furthermore, the interpolation calculation amount can be reduced simply by reducing the number of deformation steps. However, the image is not smoothly deformed and a natural video image cannot be obtained.
A second problem is that specifying corresponding control points between the first image and the second image is difficult, when the difference in the number of the control point between the first image before deformation and the second image after deformation is large. If the control points in the first image correspond to those in the second image on the one-to-one basis, interpolation calculation is performed easily. However, when the control points do not correspond on the one-to-one basis, it is necessary to perform control point synthesis processing or control point degeneration processing separately in parallel to the interpolation calculation of the control points. Therefore, the interpolation calculation amount of the control points becomes large, the processing becomes complicated, and time-cost is increased.
A third problem is that it is generally difficult to select the control points and to specify the corresponding control points between the first image before deformation and the second image after deformation. In particular, 3D morphing is more difficult than 2D morphing, because selecting the control points and specifying the corresponding control points are performed three-dimensionally, so that the processing is complicated.
Therefore, in view of the foregoing problems in the conventional morphing process, it is an object of the present invention to provide a morphing image processing system and method that can retain a large number of control points of the image and a large number of deformation steps, can maintain the image quality, can reduce the interpolation calculation amount while realizing smooth image deformation, and can reduce the load of the processing.
It is another object of the present invention to provide a morphing image processing system and method that can reduce the interpolation calculation amount while realizing smooth image deformation, and can reduce the load of the processing, even if some control points cannot find their corresponding control points in the morphing step between the first image and the second image.
It is another object of the present invention to provide a morphing image processing system and method that can automatically select the control points in the first image before deformation and the second image after a deformation and specify the corresponding control points.
In order to achieve the above objects, a morphing image processing system of the present invention for performing morphing processing for deforming a first image smoothly into a second image while forming intermediate images interpolated between the first image and the second image includes a low progression level polygon model generation processing part for performing polygon reduction processing to a first polygon model corresponding to the first image and a second polygon model corresponding to the second image to generate a first low progression level polygon model and a second low progression level polygon model having a reduced number of polygon vertices, wherein the polygon reduction processing is for obtaining polygon vertex data on each progression level and metadata, and a polygon vertex reduction process is recorded in the metadata; a low progression level polygon model morphing processing part for specifying corresponding vertices between the first low progression level polygon model and the second low progression level polygon model and forming intermediate low progression level polygon models interpolated between the first low progression level polygon model and the second low progression level polygon model; and a morphing intermediate image generation processing part for performing reproducing processing to generate intermediate images between the first and second images by using the metadata, based on the respective intermediate low progression level polygon models, wherein the reproducing processing follows procedures in the opposite order of the polygon vertex reduction process.
With the above embodiment, an image can be represented with a polygon model, and the number of polygons can be reduced by using the polygon reduction processing, so that the interpolation calculation amount can be reduced by performing interpolation between a reduced number of vertices of polygons. Furthermore, polygon models in the original progression level can be generated from the low progression level polygon models to generate intermediate images. Therefore, these intermediate images make it possible to achieve image quality of high precision and smooth deformation.
In other words, a polygon mesh before and after deformation is subdivided to have a large number of polygons, and polygon reduction processing is performed to automatically generate a polygon mesh having an arbitrary number of polygons (a reduced number). Then, interpolation calculation can be performed using the vertices of the polygons of the obtained rougher mesh. Using this polygon reduction processing reduces the number of polygon vertices while retaining the features of the outline, so that distortion of the image can be reduced.
It is preferable that the morphing image processing system includes an intermediate metadata morphing processing part for generating intermediate metadata interpolated between first metadata corresponding to the first low progression level polygon model generated in the low progression level polygon model generation processing and second metadata corresponding to the second low progression level polygon model generated in the low progression level polygon model generation processing, the intermediate metadata corresponding to respective intermediate low progression level polygon models, and that as the metadata used in the reproducing processing of the morphing intermediate image generation processing, corresponding metadata are used for respective intermediate low progression level polygon models.
With the above embodiment, intermediate metadata corresponding to respective intermediate low progression level polygon models can be generated also for metadata necessary to restore the polygon models of a higher progression level from the low progression level polygon models. Therefore, intermediate images with high precision can be generated from the intermediate low progression level polygon models.
Furthermore, in the morphing image processing system, it is preferable that a polygon model represented with a polygon mesh covered with triangle polygons is generated in the polygon model generation processing, and the triangle polygons are obtained by applying a grid with crossed lines extending in each axis direction at predetermined intervals over an image and dividing each region defined by the lines by a predetermined area into triangles having an equal area.
With the above embodiment, it is not necessary to set control points for morphing for a captured image, and a polygon model representing the captured image with a polygon mesh can be generated. Thus, morphing processing can be executed with this polygon model.
The morphing image processing system of the present invention can perform morphing image processing in which the interpolation calculation amount can be reduced by using polygon reduction processing to input images, and that allows high precision images and smooth deformation.
Furthermore, according to the morphing image processing system of the present invention, intermediate images can be generated using the intermediate metadata generated by intermediate metadata morphing that interpolates between the first metadata and the second metadata, so that intermediate images with a higher precision can be obtained. Moreover, the morphing image processing system of the present invention can generate intermediate images by utilizing the difference value of the intermediate metadata, so that the data amount of the intermediate metadata can be reduced.
Furthermore, the morphing image processing system of the present invention can specify corresponding vertices, even if there is a difference in number of the control points between the base image and the target image, by adjusting the difference in number of the vertices therebetween by interpolation.
Furthermore, the morphing image processing system of the present invention can execute morphing processing in accordance with the degree of detail of morphing for each region by designating a partial region of an important image such as the eyes.